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BACKGROUND 


¢ The National Aeronautics and Space Administration (NASA) defines risk management as an 
integrated framework, combining risk-informed decision making and continuous risk 
management to foster forward-thinking and decision making from an integrated risk 
perspective; therefore, decision makers must have access to risks outside of their own project 
to gain the knowledge that provides the integrated risk perspective. 


e Inthe Summer of 2013, through the Goddard Space Flight Center (GSFC) Flight Projects 
Directorate (FPD) Business Change Initiative (BCI), discussions began to integrate project risks 
into one repository to facilitate access to risk data between projects, the cross-cutting risk 
framework (CCRF) team was formed 


¢ The creation of the consolidated risk repository, in parallel with the initiation of monthly FPD 
risk managers and risk governance board meetings, are now providing a complete risk 
management picture spanning the entire flight projects portfolio 
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BUSINESS CHANGE INITIATIVE 


The Business Change Initiative (BCI) is a comprehensive evaluation of program planning and control 
(PP&C), as well as programmatic communication and information sharing mechanisms to improve cost, 
schedule, and overall performance across the Flight Projects Portfolio 
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Disparate Community Integrated Community 


= Rising costs and schedule delays 

= Varying processes and tools 

= Increasing competition : . 

= Oncoming retirement wave and Breaking Barriers 
corresponding knowledge gap risks 

= Expanding reviews and data requests 

= Confining pockets of PP&C excellence 


RESULTS 


Increased visibility and accountability 
Advanced tools and resources 

Common policies and requirements 
Enhanced training and knowledge sharing 
Expanded risk management 

Improved cost and schedule performance 
across mission portfolio 


= Numerous re-plans RESULTS 
cae = Efficient PP&C 
=" Redundant activities and systems 
ae : = Accurate data 
= Limited new business : Rae : 
aS : : = Consistent application of practices 
= Limited experts/diversity : : 
aahees Rene ates = Skilled, diverse workforce 
= Poor communications and visibility into cues é é ee 
? = Early risk identification and mitigation 
project performance : 


On-time delivery and cost effectiveness to 
ensure mission success 


Insufficient willingness and mechanisms to 
share expertise 
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GETTING OFF THE GROUND 


FPD sponsored the effort as part of the BCI, enlisting the help of GSFC’s Safety and Mission Assurance 
Reliability Office to define, design, develop, and deploy an approach to streamline the process of 
identifying “cross-cutting” risks 


Reduced mitigation costs by 
leveraging solutions over 
multiple projects and verify the 
consistency of selected 
strategies 


Improved lessons 
learned by looking 
across cases and 
projects for 


ae Mitigation 
commonalities Planning 


Increased data integrity Data Integrity Result: Enable executive leadership and 
through improved ae program/project managers with improved 
accessibility, insight, trend analysis, and information 


integration, and report 
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FEASIBILITY STUDY 


A cross-cutting Excel Workbook was developed, structured 
around the guidance provided in the NASA Risk Handbook. 
Historic and current risks were downloaded from select 
project risk registers. Individual risks were categorized; 

1,130 unique risks on 5 projects were used to develop a proof 
of concept. Filters were designed to quickly narrow the field 


NASA’s Cross Cutting Risks Guidance 


* Requirement 59261 
* Requirement 59262 
* Requirement 59265 
You “Shall Do” cross- 
cutting risk analysis 


Cross cutting risk analysis 
categorizes individual risks 


GSFC GPR 7120.4D 


* Projects will elevate cross- 


cutting risks up the 
management chain 


* Threat value calculation 


of risk information for detailed cross-cutting review and 
analysis to identify: 
> Tactical opportunities for improving mitigation process at 
the portfolio level tinlien 
> Strategic opportunities for identifying system ie 
improvements 


using standardized 
taxonomies 


Each risk is 
assigned an overall 
category based on 
its location in the 
pre defined 
taxonomies 


RISK 
CATEGORIZATION 


Location in Location in 
Departure Asset 
Taxonomy Taxonomy 


The results of a feasibility study assisted to identify critical next steps: 
Determine a home for the data 
Launch RM focus group meetings 


Request data from all FPD programs/projects 
Organize the data for further use 


Establish a repeatable, monthly process for the cross-cutting risk framework (CCRF) 
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APPROACH 


The data would be maintained through the creation of a monthly collection of month-end risk register 
submissions from all program/projects; a CCRF monthly process was established: 


Initial data entry was = 
completed in the FPD cross- 
cutting risk database for the 
FPD mission portfolio by the 
FPD front office; a\| project 
were organized under their 
respective division office and 
line of business; milestone 
dates were inserted from the 
FPD master schedule and MSR 
reporting 


fi FPD Risk Community of 


Practice uses data to run 
queries on risk 
information and 
formulate monthly risk 
reports for the FPD risk 
governance board 


mE ¥ 


Risk data is exported from ’ The FPD SharePoint The independent risk 
project risk registers administrator initiates lead(s) categorizes all risks 
monthly and saved ina input from the document noted inreport (new and 
Microsoft Excel format; the library into the FPD cross updated) into appropriate 
project uploads into a cutting risk database; the “departure” and “asset” 
document library as part of database recognizes all categories and saves in the 
the monthly project tag new and updated risk FPD cross-cutting risk 
up/MSR process entries to formulate a database 


report for the 


independent risk lead({s) 


meets at month-end to review 
monthly risk report from FPD 
Risk Community of Practice; 
results are shared at FPD 
Program Managers Staff and 
the top cross-cutting risks are 
reported at the last 
Wednesday Weekly Tag-up 
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alalgl 
The FPD governance board 2282) All monthly reports are archived 


and accessible by FPD 
program/project leadership; 
database accessibility remains 
open with controlled access to FPD 
leadership throughout the monthly 
process to run individual reports 
and data queries; FPD front office 
maintains the database 
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CROSS-CUTTING RISK DATABASE V.1 


Controlled Access 


National Aeronautics and Space Administration 
Goddard Space Flight Center 


¢ Identified user groups (editorial, contribute, 
and read-only) 


y é 
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Welcome to the FPD Cross Cutting Risk Analysis Database 
The database was last updated: 8/20/2015 


¢ Monitored use by user 
¢ Logged and tracked monthly updates 


1 of 1 | | 1 a 
Cross Cutting Dashboard 


Accessibility 


¢ Availability of cross-cutting risk data to FPD 
leadership and risk managers 


4233 


11988 156 


35 35 


09-20-2015 
Total Risk Records is the cumulative risks that have been collected each month 


¢ Visibility into risk management planning and 
lessons learned 


Analysis 


C - Moderate 


C-High C - Very High 


¢ Features to trend and benchmark against other 
risks, projects, and lines of business 


* Queries and reports to drill into related risks to 
identify cross-cutting trends 


¢ Dashboards to assess the current risk posture 
of the flight projects mission portfolio 
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Search this site 


Parameters 


Risks Register Month 
1/20/2015, 2/20/2015, 3/ 


States 
Closed, Open 


Divisions 
Earth Science Projects Dit 


Projects 
DSCOVR, ESMP, GOES-R 


Types 
Other/Unspecified, Progre 


Likelihood 


Apply 
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ASSET AND DEPARTURE HIERARCHY 


Data organization began with grouping by FPD division, followed by program and project; for the next 
level of risk organization, the team grouped risk data into two major categories, Assets and Departures, 
as suggested in NASA Risk Management Handbook, Section 4.3.3.1.1 Cross-Cutting Risks 


Departure Level 1 


Asset Category Level 1 


Asset 
Departure Level 2 sie tocke Asset Category Level 2 Category 
evel 2 
Level 2 
Departure |] Departure |] Departure Asset Asset Asset 
Level 3 Level 3 Level 3 Level 3 Level 3 Level 3 
EXAMPLES 
° External ° External 
° Programmatic ° Public Lives 
° Economic ° Program Wide Assets 
° Acts of God ° Personnel 
° Programmatic ° Facilities and Equipment 
° Human Resource ° NOAA 
° Design Staffing ° Other 


° Launch Support Staffing 


Mission Specific Assets 


° Subcontractor ° Space Vehicle Bus 
° Technical ° Communication 
. Design . Propulsion 
° Design V&V ° Space Vehicle Flight System 
° Reliability Design ° Launch Vehicle 


° Engineering 
° Process Control 


° Ground Segment Space Vehicle Control 
° Ground Element Data Collection 
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INITIAL FINDINGS 


Early results allowed the team to convey common risks across multiple projects. Searches on risk type 
(cost, schedule, technical) produced initial findings 


CROSS-CUTTING > DEPARTURES > EXTERNAL > POLITICAL > SEQUESTRATION/CR FUNDING 


Risk Summary Data 
> Count of risks 
> % budget at risk 
> Budgeted S at risk 
> Risk details 


Filtered by (any or all combinations) to 
Uncover Commonalities 
> 3 Levels of Departure Drill 


Down 
> 3Levels of Asset Detail Drill 


Down 

Date opened 

Project Phase Opened Lockheed Moren 
Project Name scons 


Risk state (Open, Closed) Sequestraton or 


“her Externe 
Programmat 


mstability (Schedule 


mstability (Funding 
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AUTOMATING THE SEARCH FOR CROSS-CUTTING RISKS 


¢ The “Independent Risk Team” was established to leverage all the risk data in one 
location; the Team focused on: 


¢ Automating the search for cross-cutting risks 
¢ Identifying trends, strengths, weaknesses, and opportunities 
¢ Providing monthly briefings to FPD and SMA management, and soliciting feedback 


¢ Initial analysis revealed a high percentage of risks being classified as “Other;” this caused 
the Team to re-examine the categorization approach from a “clean-slate” perspective 


¢ In parallel, commercial-off-the-shelf (COTS) data analysis software packages were 
evaluated; open-source word count software was identified and evaluated 


¢ Approach to prototype the word count function in the SharePoint CCRF tool was 
approved; a prototype was developed based on a Team brainstorming session 
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WORD COUNT FUNCTION 


The word count concept: if the user is not sure what word they need to use for their database 
search, they could view a list of words found in the database, listed in order of frequency, with the 
most common words at the top of the list 


WORD COUNT FUNCTION BRAINSTORMING WHITEBOARD 
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CHALLENGES 


Initial word count results revealed the extent of the challenge: the number of different words 
contained in the collection of project risks was overwhelming 


Since guidance to the projects had not included standard nomenclature, use of a single word may, 
or may not, return the desired results 


The Team determined that the word count function would benefit from “inclusion” and “exclusion” 
lists to return results from synonyms similar to Google searches 


Word/PhraseSearch | Results 


Integration and Test 
| Integration & Test 
| 1&T 


project integration and 


test phase JEM) 


InT 

| Integration 

| Test 

pyro valve 
ball valve 
cutoff valve 
isolation valwe 
pyrotechnic valve 
shutoff valve 
shut-off valve 
cut-off valve 
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BROWSE PAGE C) SHARE 


FPD Cross Cutting Risk Analysis Database Home 


> Word Count Report 


Dashboards 


0 - Word Count Configuration Page Configure excluded words Configure phrases/synonyms 
Jata Connections 


Libraries = Select a project: * Filter mode [+] Add cornparison project 
rpwrraaaa ha arama ares All Projects 3 Comparison mode —_[-] Remove comparison project 
PerformancePoint Content 
Include project phase data Include newest risks only Count keywords only 


Recent 


Filter by Responsible Division 


UniqueRisksReports 
UniqueRisksMaster Click on a bar or bar segment to drill down and see details. 


srihenaeitions ins To see a list of unique risk IDs that contain a word, drill down to see the word's details, and click on the word in the legend in the upper right of the graph 
Sige cece: to see risks from all projects, or on 2 project's bar to see risks only from that project. 


UniqueRisksMaster 


UniquelssuesMaster Search Clear Search 


Enter search terms (comma separated) above. 
Word Count Report << First <Prev 1of20 Next> Last >> 


Word Count Configuration 


Site Contents 


All Phases (All Projects) = 


0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 


1700 1800 


schedule 
testing 
test 
requirements 
flight 
system 
data 
design 
launch 
cost 
mission 
instrument 
project 


performance 
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REEXAMINING THE CATEGORIZATION TECHNIQUE 


Too many risks ending up in the “Other” category; the original Assets and Departures approach was 
re-examined; the categories needed to more closely mirror the GSFC workflows 


Project work breakdown structures (WBS) were compared to Asset and Departure categories; a risk 
breakdown structure based on the project’s WBS was deemed to be an improvement 


After drafting a prototype, the Team solicited feedback from other RM subject matter experts; 
through this collaborative effort, established the replacement data structure for evaluation 


The last set of new risks was chosen to be the test data for the evaluation; the Team met to 
collectively categorize the approximately 30 new risks 


The result was unexpected: the Team was having the same difficulty categorizing the risks as they 
had had when using the Asset and Departure structure 


The risks were written in a manner that allowed any analyst to bin them in more than one category; 
time after time, if there were four analysts reviewing a new risk, each of them would select a 
different category 


The Team concluded that, since there had not been any restrictions on nomenclature, the data was 
too random to categorize consistently; often the terminology used in risk titles, risk statements, and 
context statements was inconsistent 
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Tier1 Tier 2 Tier 3 Tier 4 Tier 5 Tier 6 


Radiator 
Heat Shield 
Thermal 
Interface 
Cryogenic Dewar 


Cooling System 
Cryocooler 


Connector 


Cable/Harness 


1553 


Latch 


Bearing 
Gimbal 


Spacecraft 
Deployables 


Antenna 
Transmitter/Rec 
Comm Power Amp 
Switch 
Waveguide 


Validation Testing 
Coding 
Varification Testing 


Maintenance & Sustainment 


Fabrication 
Assembly 


Interface 


Observatory/Host 
rv/ Structural 


Accoustic 
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USE OF KEY WORDS WAS EXPLORED 


It is standard practice to assign key words to configuration-controlled documents as a method of 
simplifying searches for like items; assignment of key words to the CCRF risks was evaluated 


By now, the Team was wary of adding any CCRF tool automated functionality (essentially creating a 
word exclusion list for each risk automatically during the ingest process) without first validating the 
concept; the Team tested the concept on known cross-cutting risks 


The Team removed extraneous words in each risk, then compared the correlation of remaining 
words; the expectation was a correlation above 50 percent (greater than a coin-flip) 


The test was performed on two pairs of cross-cutting risks; the first pair yielded only a 20-30 
percent word correlation and the second pair yielded a slightly /ower correlation 


When the experiment was repeated between a set of risks chosen completely at random; the 
results were almost identical to the comparison of known cross-cutting risks 


The Team concluded that this was another case of the data randomness hindering efforts to 
automate the search for cross-cutting risks 


The evaluation of COTS analysis packages fared no better, even those with “neural network” 
capabilities (using algorithms to identify underlying data relationships in a way the human brain 
would use in a manual analysis process) 
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CROSS-CUTTING RISK DATABASE V.2 


Today, there are 3,793 risks in the database with a total of 325 issues 
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Cross Cutting Risk Dashboard 
Number of Divisions Number of Projects Number of Risks Latest Update 
" 26 3846 12-19-2016 
All Projects 


Show Open risks L- Very High 
L- High 
L - Moderate 


Show Closed risks 


Show Accepted risks aise 


Show All risks L-Very Low 211 Load All risk di 


C - Very Low Cc - Low ¢ - Moderate C - High C - Very High 


Search: 


Risk ID State Originate Date Date Closed Likelihood Consequence 


* GOES-A Program 
+ GPM 
+ ICESat-2 


+ JPSS-Flight 
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RESULTS 
Nineteen risk managers participate in sending risk registers to be included in the database. By 
September 2016, we had completed 13 risk manager meetings and 13 Risk Governance Board 


meetings that discussed commonalities that appeared across program/project risks monthly and 
strategize additional improvements. 


The process has already begun using the forums to educate GSFC’s RM community about the 
benefits of: 


¥ Standard nomenclature 
Y Consistency with the risk title, risk statement, and context statement in each risk record 
Y Use of key words assigned by the program/project risk manager in each risk record 


This education has been integrated into NASA GSFC “ABC’s of Risk Management” training course, 
and risk managers are being encouraged to integrate it into their program/project-specific RM 
training material 
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LESSONS LEARNED 


¢ Itis a difficult, time consuming task to analyze very large data sets that have significant 
randomness and independent processes — it requires a phased strategy with flexibility to 
tailor to kick-start the analysis and collect a complete data set 


¢ Organizations will benefit from having all project risk data in one database, minimally ina 
single, common repository 


¢ The process of organizing data is too subjective “after-the-fact” than when it is originally 
written — key words should be assigned to each risk by the project risk manager 


¢ The development of automatic analysis reports produced minimal results due to complex 
customization — search and word count functions from standard nomenclature in project 
data does benefit analysis to seek consistency and commonalities between risks (e.g., titles, 
statements, and classifications) 


¢ Commercial software becomes increasingly ineffective as data randomness increases — tools 
should be designed around unique organizational requirements to leverage large amounts 
of RM information and assist in making effective, risk-informed decisions 


¢ RM forums remain the most effective method of identifying cross-cutting risks 
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TERMINOLOGY 


¢ Portfolio — the collection of FPD-managed missions; projects with FPD-held program 
authority, yet externally managed; projects with FPD-provided instrument contribution, yet 
externally managed; and FPD projects that are developed in-house to include spacecraft, 
instrument, or both 


¢ Portfolio-level analysis — strategic and tactical examination of the elements of FPD’s mission 
portfolio by categorizing the missions by lines of business (grouping like projects and 
programs) 


¢ Cross-cutting risk — an individual risk with attributes and impacts found in multiple levels of 
the organization, or in multiple organizations within the same level 


¢ Departure — a statement about what might occur at a future time, a possible change from the 
(Agency, program, project, or activity) baseline project plan; it is the uncertainty in the 
occurrence or non-occurrence of the departure that is the initially identified source of a risk 


¢ Asset —an element of the organizational unit portfolio, analogous to a work breakdown 
structure (WBS); assets represent the primary resource that is affected by the individual risk 
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